Allow interaction with ray client service via Route from outside of OCP cluster #100

tedhtchang · 2023-05-11T00:35:38Z

How to test:

# clone the branch
git clone --single-branch --branch ray-loca-interaction https://github.com/tedhtchang/codeflare-sdk.git
cd codeflare-sdk/

# create python virtual env with [pyenv](https://github.com/pyenv/pyenv#automatic-installer)
pyenv install 3.8.13
pyenv virtualenv 3.8.13 localinter
pyenv local localinter

# Install python packages
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install jupyterlab
pip install -e .

# Run a sameple noteook and then Run -> Run all cells
jupyter lab demo-notebooks/interactive/local_interactive.ipynb

# clean up
pyenv uninstall localinter
# delete the cloned repo

Expect something

tedhtchang · 2023-05-31T22:04:04Z

@MichaelClifford I passed all the tests. looks good. I also renamed the example notebook.

MichaelClifford · 2023-06-05T20:26:06Z

Thanks for adding more tests @tedhtchang

LGTM

jbusche

LGTM, thank you Ted!
Note, for testing, there were two items that were different from what was listed:

Had to install pip install -r requirements.txt in addition to the requirements-dev.txt in order to get openshift installed in my pyenv
The notebook file is named local_interactive.ipynb

jbusche · 2023-06-19T04:56:59Z

@tedhtchang, with the latest code with the Ray Operator v0.5.0, the worker is now able to start all 3 init containers and that's working ok. But when I try to do the import ray step I'm getting:

File ~/.pyenv/versions/3.8.13/envs/localinter/lib/python3.8/site-packages/ray/util/client/worker.py:260, in Worker._connect_channel(self, reconnecting)
    252 if log_once("ray_client_security_groups"):
    253     warnings.warn(
    254         "Ray Client connection timed out. Ensure that "
    255         "the Ray Client port on the head node is reachable "
   (...)
    258         "more information."
    259     )
--> 260 raise ConnectionError("ray client connection timeout")

ConnectionError: ray client connection timeout

tedhtchang · 2023-06-19T19:54:38Z

Tested on my own mac and another Ubuntu VM from scratch. Both env seems to work. @MichaelClifford @Maxusmusti Could you test again?

jbusche · 2023-06-20T16:08:22Z

I tried one last time last night and local interaction worked on my cluster to default namespace from my Mac. I'm not certain why it worked this time, but perhaps a new cluster and/or some local changes on my Mac... not sure.
Was also trying to test from an Ubuntu image, but I ran out of time trying to setup my Ubuntu with the proper prereqs. Last thing Ted wanted me to try was a different namespace other than default. Will post update here once my cluster rebuilds and I can test.

jbusche · 2023-06-20T18:41:56Z

OK - tried on the IBM network with both default and jim namespaces - looked good!

jbusche

LGTM Ted, passing the tests in two namespaces
Thank you!

jbusche

LGTM
Tested in both default and non-default namespaces... looks good.

Maxusmusti

LGTM

MichaelClifford

LGTM!

Tested PR on cluster with KubeRay v0.5.0 and works as expected.

Maxusmusti

Looks good! Just two quick comments before I merge

src/codeflare_sdk/utils/generate_yaml.py

Maxusmusti

LGTM

tedhtchang force-pushed the ray-loca-interaction branch 2 times, most recently from 5f65ec2 to d6175c4 Compare May 13, 2023 00:29

MichaelClifford force-pushed the ray-loca-interaction branch from dbd8fc8 to c509066 Compare May 19, 2023 17:28

tedhtchang requested a review from Maxusmusti May 31, 2023 22:53

tedhtchang changed the title ~~[WIP] Allow interaction with ray client service via Route from outside of OCP cluster~~ Allow interaction with ray client service via Route from outside of OCP cluster Jun 6, 2023

jbusche previously approved these changes Jun 13, 2023

View reviewed changes

tedhtchang and others added 7 commits June 13, 2023 17:33

Allow interation with rayclient via route from outside of OCP cluster

a56df94

update rayclient route generation

327cb26

update test-case yamls for client route

c5951c1

add unit tests for generate_cert.py

4881ecf

Update local interactive notebook

2b1863a

Add more cert test cases

723eb3d

Replace openshift client with k8s python client

bd0934a

tedhtchang dismissed jbusche’s stale review via bd0934a June 14, 2023 17:11

tedhtchang force-pushed the ray-loca-interaction branch from dbd5778 to bd0934a Compare June 14, 2023 17:11

tedhtchang requested a review from jbusche June 14, 2023 17:14

Update initcontainer to work the KubeRay v0.5.0

aa43051

Allow rayCluster to spin up in other ns

a762019

jbusche previously approved these changes Jun 20, 2023

View reviewed changes

fix merge conflict

a3e4f0a

tedhtchang dismissed jbusche’s stale review via a3e4f0a June 21, 2023 22:40

jbusche previously approved these changes Jun 21, 2023

View reviewed changes

Maxusmusti previously approved these changes Jun 22, 2023

View reviewed changes

Add auth cell to the notebook

deeb62c

tedhtchang dismissed stale reviews from Maxusmusti and jbusche via deeb62c June 22, 2023 18:25

MichaelClifford previously approved these changes Jun 22, 2023

View reviewed changes

tedhtchang mentioned this pull request Jun 22, 2023

Create local_client_url() cluster method to return the local client url #173

Open

Merge remote-tracking branch 'upstream/main' into ray-loca-interaction

c6aca74

Maxusmusti reviewed Jun 22, 2023

View reviewed changes

src/codeflare_sdk/utils/generate_yaml.py Show resolved Hide resolved

src/codeflare_sdk/utils/generate_yaml.py Outdated Show resolved Hide resolved

remove non function method

e77b274

tedhtchang dismissed MichaelClifford’s stale review via e77b274 June 22, 2023 19:10

Maxusmusti approved these changes Jun 22, 2023

View reviewed changes

Maxusmusti merged commit df48547 into project-codeflare:main Jun 22, 2023

MichaelClifford mentioned this pull request Jun 23, 2023

Connect to an interactive Ray session from outside the cluster #53

Closed

tedhtchang mentioned this pull request Sep 12, 2023

Improve SDK and Notebook user experience of ray cluster local interaction #241

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow interaction with ray client service via Route from outside of OCP cluster #100

Allow interaction with ray client service via Route from outside of OCP cluster #100

tedhtchang commented May 11, 2023 •

edited

Loading

tedhtchang commented May 31, 2023

MichaelClifford commented Jun 5, 2023

jbusche left a comment

jbusche commented Jun 19, 2023

tedhtchang commented Jun 19, 2023

jbusche commented Jun 20, 2023

jbusche commented Jun 20, 2023

jbusche left a comment

jbusche left a comment

Maxusmusti left a comment

MichaelClifford left a comment

Maxusmusti left a comment

Maxusmusti left a comment

Allow interaction with ray client service via Route from outside of OCP cluster #100

Allow interaction with ray client service via Route from outside of OCP cluster #100

Conversation

tedhtchang commented May 11, 2023 • edited Loading

tedhtchang commented May 31, 2023

MichaelClifford commented Jun 5, 2023

jbusche left a comment

Choose a reason for hiding this comment

jbusche commented Jun 19, 2023

tedhtchang commented Jun 19, 2023

jbusche commented Jun 20, 2023

jbusche commented Jun 20, 2023

jbusche left a comment

Choose a reason for hiding this comment

jbusche left a comment

Choose a reason for hiding this comment

Maxusmusti left a comment

Choose a reason for hiding this comment

MichaelClifford left a comment

Choose a reason for hiding this comment

Maxusmusti left a comment

Choose a reason for hiding this comment

Maxusmusti left a comment

Choose a reason for hiding this comment

tedhtchang commented May 11, 2023 •

edited

Loading